\section{Insuring Politeness}
\label{sec:rob}
To insure that our search application is polite to the pages it crawls, a class for reading the pages \textit{robots.txt} and insuring that the program is polite by following what is written the in \textit{robots.txt}.

The \textit{Robots} class is responsible for fetching and storing the \textit{robots.txt} information of a web domain. This is done through the method, \textit{parseRobots}, that takes one string as input parameter. The domain url is parsed in this input parameter which then extracts the \textit{robots.txt}.

An example of a rule in a robots.txt is "Disallow: *" which means that the crawler is not allowed to crawl any pages.

Another perspective of politeness is the implicit politeness which is to avoid hitting the server too ofen. Therefore a limit has been imposed such that a domain is at most  crawled one time pr. second. 